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Background / Context: 

As described in the background for Paper 1, the consequences of failing core academic courses 
during the first year are dire. In Chicago, over a quarter of students fail at least one semester of 
algebra in their ninth grade year, and only 13% of students who fail both semesters of Algebra I 
in ninth grade graduate in 4 years. 

Offering credit recovery options is one strategy to deal with high failure rates. The primary goal 
of credit recovery programs is to give students an opportunity to retake classes that they failed in 
an effort to get them back on track and keep them in school (Watson & Gemin, 2008). It makes 
theoretical sense to try to get students to recover their algebra credits early, in the summer after 
ninth grade — before they take geometry or Algebra II and chemistry, and to put them back on 
track towards graduation. But there is little evidence about the extent to which expanding credit 
recovery leads to substantive improvements in student progression and outcomes. While it seems 
like a good idea, the pay-off may not actually be large for a number of reasons: few students who 
failed in the prior year may show up in the summer for credit recovery; few students may pass 
even if they do show up; and the gains of attending summer school for learning and for credit 
accumulation may be very small compared to students’ initial deficits in skills or the number of 
total credits they eventually need to recover. Thus, schools might put in substantial effort to hire 
teachers and find facilities for credit recovery with little pay-off in terms of their schools’ on- 
track rates, test scores, and later graduation rates. 

Purpose / Objective / Research Question / Focus of Study: 

This study examines the benefits of offering expanded credit recovery options for ninth grade 
algebra, relative to business as usual (i.e., the summer programming schools would offer in the 
absence of efforts to expand credit recovery). Evidence of the effects of credit recovery for 
getting students back on track for graduation is lacking, and this study presents the opportunity to 
generate such evidence. 

Setting: 

The setting will be neighborhood high schools in Chicago in 2008-2012. CPS is the third-largest 
U.S. district, serving more than 404,151 students in 681 schools. For this study, we include all 
neighborhood high schools (no charter or selective enrollment schools), excluding a group of 
schools that were participating in another district-sponsored algebra program. The district is 
87% low-income and 42% African American and 44% Latino. 

Population / Participants / Subjects: 

The population consists of all first-time ninth grade students who entered regular neighborhood 
high schools between 2008 and 2012. On average, there were 20,292 students in each cohort. 

For some analyses the population is only students who failed second semester algebra (Algebra 
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IB), of which there were 3,820 in 20 1 1 . All neighborhood high schools that were not in the 
excluded for the reasons described below are included for analysis (86 schools). 

In this study, some high schools in CPS received funding to implement at least two Algebra I 
credit recovery courses during the summer sessions of 201 1 and 2012 — at least one online and 
one face-to-face section. Fifteen schools participated in 201 1; in total they offered 18 pairs of 
sections (36 total). Thirteen schools participated in 2012; in total they offered 20 pairs of 
sections (40 total) (see paper 1 for details). 

Fifteen CPS high schools participated in this project in summer 2011. The participating schools 
were similar to other schools in CPS in most respects, with a few notable differences. They were 
larger schools than typical; they enrolled 1,785 students, on average, compared with an average 
enrollment of 729 students in high schools across CPS. This difference was expected because we 
could implement the study only in schools that were large enough to have sufficient students to 
fill at least two sections of second-semester algebra over the summer. The study schools also 
disproportionately served more Latino students and fewer African-American students than the 
district as a whole (see Figure 3 in Appendix B). 

Intervention / Program / Practice: 

The extra funding schools received as part of participating in this study increased their capacity 
to provide algebra credit recovery over the summer, so that all ninth grade students who wanted 
to recover the credit over the summer could do so. To boost enrollments in summer algebra 
credit recovery, a set of outreach activities were implemented in each school to encourage ninth 
grade students who failed the second semester of Algebra I to attempt to recover the credit in the 
summer. This included presentations to groups of failing students by study team members, calls 
to students’ homes, and letters sent home about the importance of passing algebra and recovering 
credits. We define the presence of the extra online course, plus the push to motivate students to 
enroll in credit recovery, as “expanded credit recovery options.” 

Schools were invited to participate in the study based on two main criteria. The primary 
criterion was the number of students who failed second semester algebra in the prior year. 
Researchers ranked schools by the number of students who failed in the prior year and 
considered those with the most failures in prior year eligible to be invited to participate. Schools 
could have large numbers of algebra failures because they had high failure rates, or because they 
were large schools, or both. The second criterion was that they were not closed for construction 
over the summer, and could potentially offer summer school. Each summer, a number of schools 
were closed for construction. We excluded charter schools from consideration because their 
students’ transcripts do not appear in the centralized data system. We also did not consider 
schools that were in one area of the district whose area chief had refused participation because of 
other algebra-related initiatives being implemented. 

Research Design, Data Collection and Analysis: 

Schools were eligible for the study if they had enough ninth-grade students who failed Algebra 
IB to satisfy the study design (40 students per school, 20 in each condition). To select schools, all 
CPS neighborhood high schools were rank-ordered by the number of ninth-grade Algebra IB 
failures in 2010, one year before the year-1 implementation, and recruited according to their 
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position in the list. If a school could not participate due to construction during summer, or did 
not want to participate in the study, the next school on the list was recruited for the study, until a 
sample of 20 sections of each condition was reached. 

This study conducts two types of analysis, using difference-in-difference approach where there 
are multiple embedded comparisons. We model outcome differences between two pre- 
intervention cohorts (2009 and 2010) and two post-intervention cohorts (2011 and 2012). The 
outcome variables include recovery rates in Algebra IB, on-track for 10 th grade rates, and 10 th 
grade PLAN scores. Outcomes for the school-level analysis include: (1) recovery rates (among 
students who failed algebra I); (2) percentage of students starting their second year of high 
school with full credits in algebra; (3) 9 th grade on-track rates at the end of summer; and (4) math 
scores on the PLAN exam taken at the beginning of October of 10 th grade. Outcomes for the 
student-level analysis include: (1) recovered algebra credit; (2) on-track at the beginning of the 
second year of high school; and (3) math score on the PLAN exam. 

The school-level analysis examines the intent-to-treat (ITT) effect, that is, the effect of receiving 
an invitation to participate in summer credit recovery. This analysis exploits the fact that 
invitation was determined by the number of failures in 2010. Thus, we compare differences in 
the outcomes from pre- to post-intervention years between schools above the threshold and 
schools below the threshold. We include failure rates and school size in the current year as 
control variables. Additional control variables include students’ background characteristics— 
students’ math scores on standardized tests at the end of eighth grade and beginning of ninth 
grade, number of semester courses failed in ninth grade, whether they failed 0, 1 or two 
semesters of algebra, race, gender, SES, and school fixed effects. 

The student-level analysis examines the effect of attending summer algebra credit recovery 
course where attendance is defined as receiving a summer grade in algebra IB (i.e., those who 
did not attend summer classes or who dropped without a grade are counted as not attending, 
receiving a grade of “F” or higher count as attending summer school) using an instrumental 
variables approach. We use district records, rather than our own records for schools that 
participated in the study for consistency. We estimate the compliers average treatment effects by 
taking advantage of the two known exogenous sources of variation affecting attendance in 
Algebra IB in the summer. One source of variation was whether schools received an invitation to 
participate on the basis of the number of failures in 2010. School-by-year interaction terms can 
be used as instruments to estimate the effect of program participation among those who are 
induced to participate. The year-by-school interactions affect the outcome through their effect on 
program participation, while the year fixed effects capture other historical events that are 
assumed to be the same for the treated and comparison schools. A second source of variation was 
whether or not the school was under construction, which was decided by the district 
independently from this study. Interaction terms between construction status and year provide an 
additional instrument to model participation in expanded credit recovery. We use additional 
control variables as described above. 

Findings / Results: 

We have just received students’ transcript data and test scores through 201 1-12 from the district, 
and are currently preparing the data for analysis. In addition, we should receive students’ PLAN 
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scores after they take the exam in October. All analyses will be completed by January 2013, 
well in time for presenting at the spring SREE meeting. 

From the first year of implementation, it looks as if schools that participated in the study 
substantially increased their credit recovery rates. Participating schools nearly doubled their 
second semester Algebra I credit recovery rate in the first year in which the study was conducted, 
compared to the prior year — from 12% in 2010 to 23% in 2011 — by offering expanded Algebra I 
credit recovery options to students. 

This preliminary finding suggests that participation in this study did significantly expand access 
to credit recovery options. It sets the stage for the analyses of the effects of expanding credit 
recovery on school- and student-level outcomes that will be reported in full in this paper. 

Conclusions: 

The conclusions (and ensuing discussion with participants) will be informed by the forthcoming 
results of the study. 


SREE Spring 2013 Conference Abstract Template 



Appendix A. References 

Allensworth, E., & Easton, J. (2005). The on-track indicator as a predictor of high school 
graduation. Chicago: Consortium on Chicago School Research. 

Allensworth, E., & Easton, J. Q. (2007). What matters for staying on-track and graduating in 
Chicago Public High Schools: A close look at course grades, failures and attendance in 
the freshman year. Chicago: Consortium on Chicago School Research. 

Allensworth, E., Correa, M., & Ponisciak, S. (2008). From high school to the future: ACT 

Preparation-Too much, too late. Why ACT scores are low in Chicago and what it means 
for schools. Chicago: Consortium on Chicago School Research. 

Archambault, L., Diamond, D., Coffey, M., Foures-Aalbu, D., Richardson, J., Zygouris-Coe, V., 
Brown, R., & Cavanaugh, C. (2010). INACOL Research Committee Issues Brief: An 
exploration of at-risk learners and online education. Vienna, VA: International 
Association for K-12 Online Learning. 

Bloom, H. S., Richburg-Hayes, L., & Black, A. R. (2007). Using covariates to improve precision 
for studies that randomize schools to evaluate educational interventions. Educational 
Evaluation and Policy Analysis, 29(1), 30-59. 

Cooper, H., Charlton, K., Valentine, J. C., & Muhlenbruck, L. (2000). Making the most of 

summer school: A meta-analytic and narrative review. Monographs of the Society for 
Research in Child Development, (55(1), 1-118. 

Denton, C. A., Solari, E. J., Ciancio, D. J., Hecht, S. A., & Swank, P. R. (2010). A pilot study of 
a kindergarten summer school reading program in high-poverty urban schools. The 
Elementary School Journal, 110(A), 423-439. 

Dynarski, M., Clarke, L., Cobb, B., Finn, J., Rumberger, R., & Smink, J. (2008). Dropout 

prevention: A practice guide (NCEE 2008-4025). Washington, DC: U.S. Department of 
Education, Institute of Education Sciences, National Center for Education Evaluation and 
Regional Assistance. 

Greaves, T. W., & Hayes, J. (2008). America ’s digital schools 2008: Six trends to watch. 
Encinitas, CA: The Greaves Group. 

Heppen, J. B., & Therriault, S. B. (2008). Developing early warning systems to identify potential 
high school dropouts. Washington, DC: American Institutes for Research, National High 
School Center. 

Jerald, C. (2006). Identifying potential dropouts: Key lessons for building an early warning data 
system. Washington, DC: Achieve, Inc. 

Kahne, J. E., Sporte, S. E., de la Torre, M., & Easton, J. Q. (2006). Small high schools on a 
larger scale: The first three years of the Chicago High School Redesign Initiative. 


SREE Spring 2013 Conference Abstract Template 



Chicago: Consortium on Chicago School Research. 

Lee, V.E. & Smith, J.B. (1999). Social support and achievement for young adolescents in 
Chicago: The role of school academic press. American Educational Research Journal, 
36(4), 907-945. 

Matsudaira, J. D. (2008). Mandatory summer school and student achievement. Journal of 
Econometrics, 142, 829-850. 

Newmann, F. M. and Associates (1996). Authentic achievement: Restructuring schools for 
intellectual quality. San Francisco, Jossey-Bass. 

Raudenbush, S. W., Martinez, A. and Spybrook J. (2007, March). Strategies for improving 
precision in group-randomized experiments. Educational Evaluation and Policy 
Analysis, (29)1, 5-29. 

Roderick, M., Engel, M., Nagaoka, J. (2003). Ending social promotion: Results from Summer 
Bridge. Chicago: Consortium on Chicago School Research. 

Slavin, R. E., & Madden, N. E. (1989). What works for students at risk: A research synthesis. 
Educational Leadership , 46, 4-13. 

Watson, J., & Gemin, B. (2008, June). Promising practices in online learning: Using online 

learning for at-risk students and credit recovery. Vienna, VA: North American Council 
For Online Learning. 

Zvoch, K., & Stevens, J. J. (2011). Summer school and summer learning: An examination of the 
short- and longer-tenn changes in student literacy. Early Education & Development, 
22(4), 649-675. 


SREE Spring 2013 Conference Abstract Template 



Appendix B. Tables and Figures 


Figure 1. Theory of Action Behind Summer Online Algebra Credit Recovery 



Dashed lines indicate a negative relationship or disruption in the path. 


Table 1: Number and Percentage of Students Per Condition by Block, Summer 2011 




Passed Algebra IA 

Failed Algebra 1A 

Algebra IA Status 
Unknown 

Total 

Condition 

Gender 

Number 

Percent of 
Students by 
Condition 

Number 

Percent of 
Students by 
Condition 

Number 

Percent of 
Students by 
Condition 


F2F 

Female 

44 

15% 

28 

10% 

31 

11% 

103 


Male 

70 

24% 

58 

20% 

59 

20% 

187 


Total 

114 

39% 

86 

30% 

90 

31% 

290 

Online 

Female 

45 

15% 

37 

12% 

35 

11% 

117 


Male 

73 

24% 

60 

20% 

56 

18% 

189 


Total 

118 

39% 

97 

32% 

91 

30% 

306 


Table 2: Number and Percentage of Students Per Condition by Block, Summer 2012 


Passed Algebra IA 

Failed Algebra IA 

Algebra IA Status 
Unknown 

Total 

Condition Gender 

Number 

Percent of 
Students by 
Condition 

Number 

Percent of 
Students by 
Condition 

Number 

Percent of 
Students by 
Condition 


F2F Female 

56 

14% 

52 

13% 

41 

10% 

149 

Male 

83 

21% 

95 

24% 

70 

18% 

248 

Total 

139 

35% 

147 

37% 

111 

28% 

397 
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Online 

Female 

53 

13% 

55 

14% 

44 

11% 

152 


Male 

81 

21% 

93 

24% 

69 

17% 

243 


Total 

134 

34% 

148 

37% 

113 

29% 

395 


Figure 2. Credit Recovery Course Grades by Condition for Cohort 1 
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Figure 3. Characteristics of Study Schools and CPS High Schools 



SREE Spring 2013 Conference Abstract Template 



